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1.0 summary ; a statistical comparison was made between two 
^utomat^d devices which were used to count words, sentences 
andi syllables (data points which)are needed in the Flesch - 
;Reading Ease Score to determine^he reading grade level of 
written material) , The devices are the Automated Flesch ' 
Count (AFC) developed by Kincaid and McDaniel and the Navy 
Automated Counter (NAC) developed by the Navy to c^^Qduct^a 
readability study of rate training and correspondence - 
manuals. Comparisons were made between the speed and 
reliability jof the automated devices, and between the speed" 
and reliability of each device and th^e manuaX method of 
CQunting data points ♦ "* / ' ^ 

* 

• Similar reliabilities were found f or >ot> of the 
devices and for the manual method. Counts made using the ^ 
device^^ ht)weyer, were done almost twice as fast as oQuntg 
done manually The NAC was 20% faster than the AFcy-but,. 
this differ-ence may be largely attributed ta>a'^pr^'0tii|e .'./\ 
advantage which the subjects using the NAC Jia;d^.aVieF'--'''ii^e ;^- 
subjects using the AFC* Nevertheless, for art^/i^'^|id4^^^ 
study whicli involves detailed counting of wqx<l^ ^"'^-^entd^ 



syllables, or similar data points/ use of an autojnaidted; 
device such as the. NAC or -AFC is highly recommended as,- 
tiie most cost-effective technique available., " ' 



2.0 ACKNOWLEDGEMENTS . This evaluation could not have been 



^ undertaken without those people who patiently^served as 
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data points manually after speTiding two morilfhs us in g V,hC' 
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tistical 'analysis ; Mi-. , w. MQDaniel, who prpvided Gomparison 
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' " ' ' . " -' ■ ■ 
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, 3. 0>- PURPOSE.. The, purpose, of this report is to . present the 
_ results of a comparison between two automated ^devices which 

were used to courit words, sentences,, and syllables... These- 
, data points (words, sentences, and syllables) are needed in 

^ome r.&^dxng formulas for determining the reading grad^'^ 
'■ level of written' m^.tgrial^... 



•4. 0 BACKGROUND-.- The' "Chief of , Naval .pducation and Training 
Support (C.NETS) recently initiated, a study . to 'determijife the 



^.^-ading.. g r ade^eve g . g f , . ^ .3t Jl .R at . g .,..T.r aA n ln g ;M g i.auaJ^!^ RT M si; a nd. . , 

, Non-R^s,iden.t Career Cours.e5,..iIJRCCsli^ee.JRe£erence^aU.-,.irx>...-.^ 

was formed 

After analyzing several measures of readability, the Read- 
ability Wbrking Group chose the Flesch Reading Ease bcore 
(see Reference 2) as the most valid and feasible measure to 

^ use. The Flesch Reading Ease Score required that a sample 

* 

cjft 25 or more parages be selected from each NRCC and RTM. 
The passages wefie from 100* to 150 words in length, and a 
count had to. be \iade of the number of words, sentences and 
syllables in each passage. Over 14,000 sample passages from 
188 NRCCs and 185 RTMs were counted during the course of the 
study. At the initial^ stage; of; this study the Readability 
Working Group decided tha't^ using', k manual method (i.e., a 
pointer to track position and summing the count by mempry) " 
' 'wimJs^^e highly unreliable and extremely tedious for this 
.numbei: of passages. A search of the literature was con- 
ducted to find, a better method, and two possibilities were 
found. The first, was the Automated Readability Index (ARI) 
" developed by Smith and Senter (see Reference 5) . The ARI 
consisted pf a modified electric typewriter connected tp a 
tabulator which recorded the ijtonber of" strokes (letters), 
words and sentences in a sample passage as the passage' was 



typed. ' Xverage w6rd and average sentence length could 



' be calculated, and these data used in a regression equation 



then 



' ( . ' 

to obtain reading grad'e level. The second method- was the 
-AutoTn^te^nFT.escn count (AFC) {see Reference 3K The^AFC~ 



- -tr§fe-a-a- simpTe appara^s""-'"ah^^ 

ronome --"to count data points " (syllaMes7~words~"and~ 
sentences). These data points were counted in synchrony 
with a metronome which actuated the counter. A manually, 
^ntrolled switch started and stopped the counter at the 
beginning and end of each passage. A description- of the AFC 
may be found in Appendix A. ^ 

Some difficulties were associated with both of the 
above methods. Although the ARI was highly reliable and 
valid, it w^s not as cost-effective as the AFC. It would 
have b§en necessary to purchase and modify several electric 
typewriters, and only specially qualified typists could have 
operated the machines, ^e^ major disadvantage of the AFC 
was that the personnel who would be doing the counting >would 
be in the same room, .and the noise? of eight to/ twelve 
metronomes operating -simultaneously may have /been too 
distracting to obtain a reliable count. 

A third alternative, devised by the^NETS -Readability 

^Wprking Group, was based^ on the AFC principle. The met- 
ronome was replaced by a stylus with a prjessure-sensitive 
Switch. The stylus wa^ designed like a pencil so th^t it 
could be pointed at each ;syllable, word, and sen^^^^^ be 
counted. Pressing the st^^lus against the page activated an 

9 
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electro-mechanical counter which totalled each of the data 

points. The specifications for this-devi0eT-whi<>h^-has -been 

— — .^^ — ^ — _ . . . , 

AgJPQd J:he_^ Automated Counter (NAC) , can be Jound dn 
Appendix B. The NAC was easy tp operate, simple ^o con- 
struct, quiet, and inexpensive. 

Previous results^j^yi^ained by McDaniel (see Reference 4) 
have compared 'the AFC with mantial methods of 'counting words, 
sentences, and syllables. McDaniel used eight college 
Students to couht data points^ both manually and with 'the 
AFC. The s^ample passages to be counted were twenty para- 
graphs of tKe "Minnesota Reading Examination for College 
Students Forms A ^nd B. The number of sfeconds taken to 
count the three sets of data points in each passage were 
recorded, and reliabilities (determined through interanalyst 
coefficients) were compute^. Reliabilities fo;r the three 
sets of data points Were high (over .89) for both the manual 
and AFC methods. McDaniel^ found that the AFC was sign.i- 
ficantly faster than the -njanual method in cQunting syllables. 
The AFC and the manual met^iod were about the same, however > 
in the number of seconds taken to count words and §fen-tences. . 

A determination was- therefore made of the speed and 
reliabilij:y of the NAC, The NAC wa^ compared to-^b^th the 
manual and AFC methods. The, results of this comj^arison are 
presented in the following paragraphs. 

5' 



5.0 METHOD * The following personnel, procedures, ^nd data 
we^^ used or_ coii^Giad^^iuring^ ..tiais jcojnparxson 




Si 1 S ub j e c - t - P Q r s o n neJr ^ This subject parsonngl whn r\ir\ 

/ • 

the counting ""were tTie s~ame personnel who co^ireG/ced the Tta'tra 

points for the CNETS readability study mentiarled previously 

in paragraph 4.0. They included 15 females /4nd one male who 

were selected from 33 applicants by means pf a structured 

screening process, including a reading skill .test. The mean 

a^e of the subject ^ersonne^ w^s 32.3 years (ranging from 18 

to 56 yearsl. Formal : education ranged/frl^m 12 to 18 years 

/ • 

(averaging 14.5 years). The edjicatiqh level of this group ^ 
was comparable to the subjects used^*/£y McDaniel. 

5.2 Procedures . Because these subject personnel were 
already committed to the CNETS readability study, it was 
^decided that a study similar to that conducted by McDaniel 
was not feasible. The analysis was therefore shortened by 
having these pes.onnel count only ten passages (Form A) of 
the "Minnesota Reading Examination for College Students" 
instead of the 20 passages used by McDaniel. The subject , 
personnel were asked to do theSd <2ounts during breaks in the 
readability study. These personnel had already practiced ^ . 
the NAC method for about two months during the readability 
Study. In addition, they had previously counted the RTM and 
NRCC materials for about, three days using the manual m0,thod. 
They were, therefore, well practiced in both methods. 



The sixteen subject personnel' w^re randbinly divided , 

x?rtQjbwQ~gi^u]^^ 1- B_a,aO:rLiiLl7_3SZ^ 

two groups had comparable counting skills for the manual and 

NAC"^ methods , both groups counted passage I manually ^nd 

passage 11^^ using the NAC. These counting methods were then 

alternated for each group for passages III 'through X. Group 

1-8 counted odd-numbered, passages maitlially ancJ^ ev^-numbered 

passages automatically. Groups 9-16 counted odd-nuitibered 

passages automatipally and even-numbered passages manually. 

The syllables in each passage were counted first, followed 

by a count of wordy^nd then a count of sentences. Table 1 

. • ' • -I 
, presents a graphic summary of the testing procedure us^d for 

each groilp. /' , " v ^ 

Since only hair of the sample passages used^ by McDaniel 
were used in this study, a new analysis of the McDaniel ^ata 
had to be'mad^. The original data were subsequently ob- 
; tained for/4even of the eight McDaniel subject personnel. 
These da,f a included the -number of ^ seconds ta,ken by the o 
McDapieJ. group to count the syllables, words and sentences. 
yd the ten sample passages using the*manual and APC^thods. 
A direct comparison of data tiould therefore be madeV between • 
tj|fie McDaniel group'^ and the present group. ■ - \- ^ 

i<3 Statistical An'alyses . The following comparisons 
were tested Tf or significant' differences using conventional 
t-tests (p .05; two-tailed) ; , , t . 



a. Mean reliability of counting words, sentences 
and syllables between Navy groups 1-8 and 9-16 for passages 
I (manual metiiod) and II '(NAC) . Reliability was determined 
using* the standard error of the mean. 

• b. Mean sreliability between (1) the manual • 
method and each of tl\e automated methods '(NAC and AFC 'K and 
(2) between the UhC and AFC, Again, reliability was deter- 
mined using the standard error of the mean. 

c. Mean speed between the manual and automated' ' 
methods, #or tiig^ McDaniel group. V 

d. Mean speed between the manual and automated 
methods for the Navy group (groups 1-8 and 9-16 combined). 

e. Mean speed of the manual method' teetween the 
Navy group and the McDaniel group. ^ 

^ r^'TIean speed of the automated methods for the 

Navy group and' the" McDaniel group. ' * ' 



i 



6.0 RESULTS . The following paragraphs present the results 
of these analyses. 

6 . 1 Comparison of Navy Groups 1-8 and 9-16 for 

* 

Reliability . ReliajDilities between (groups 1-8 "and 9-16 on 

passage,! (which was done by both groups manually) and 

passage II (which was done by bot^ groups using the N^C) are 

* 

presented on Table 2. The- reliabilities of the two groups 
in 'Counting the data points manually and with^the NAC were 

8 . ' 



found to be similar. These findings indicate that the two 
groups wdre counting the data points in a similar fashion 
for both techniques (manual and NAC) . Because of these 
similarities, the two groups will be referred to as a single 
Navy group, 

6.2 Comparison of Reliability Among Methods . This 
comparison was made using passages III through X as counted 
by the Na^ and McDaniel groups. The reliabilities of the 
automated methods (AFC and NAC) in counting data points were 
found to be similar to each other, as well as similar to thg 
manual method. This finding replicates the results found 
earlier by McDaniel (see reference 4) . 

6 . 3 Comparison of Speed Between Manual, and Automated 

Methods. TaJ^le 3" presents the number seconds taken to 
— * k 

perform the |3ounts using tlte manual and automated methods* 
ForSythe Navy group, syllables, words, and total count (the 
totd3, number of seconds taken to count syllables, words, and 
sentences combin^,d) were performed significantly faster by 
the NAC than by the mknual method. Speed for counting 
sentences was the same using both methods* The McDaniel 
group performed s^llailes and total counts faster w^^ the 
AFC than the manual method, but counting words and s^tences 
was similar tor both the AFC ^nd manual methods* . 

6,4 Comparison Between the Navy Group aad the McDaniel 
Group for Speed of Manual Methods . A determination was made 

• '14 



as to whether the two gi;oups were equally fast in manually 
counting words, sentences, and syllables. Statistics for 
this comparison are shown on Table 4 for passages I, 
and III through X combineci. The Navy group was found to 
manually count syllables' significantly faster than the 
McDaniel group (an average of nearly 1^ minute and 2p seconds 
faster for each passage). This difference was so large that 
total count was significantly faster as well. Speed for 
counting words and Sentences was, however, similar for the 
tvfo groups . 

6. 5 Comparison Be€weei^ the Navy Group and the McDaniel 
Group for Speed of Automated Methods . A determination was 
also made of whether the AFC and the NAC. methods differed in 
speed. Statistics for the two groups on passages II through 
X combined are shown on Table 5. The data in Table 5 show 
that the NAC was significantly faster ^han the AFC for 
counting syllables, words, and total count. The AFC* was 
signifp.canjj:ly faster than the NAC for counting sentences. 
It must be remembered, however, that the Navy group also 
counted^^syllables 'and total count faster manually than the 
McDaniel group, and this effect may have made for faster 
automated speeds. 

7.0 DISCUSSION . The results show that the McDanieJ. and the 
Navy groups ^were equally reliable in counting the' data 

10 



points manually. It was also tounU'^Vat the counts from the 
two devices were equally and ^igiily i<^§liable, and comparable 



in reliability to the manual method<_ ,The two groups were 
also compared on the speed with which data points were 
manually counted. If the groups; 3vere equally* fast in 
counting manually, then differences in'speed using the 
automated methods would fte the result of the devices and not 
of the subject personnel. It was found, however, that the 
Navy group was similar to the McEfenieTgroup in manually 
counting word and sentences, but the Navy groups manually 
counted syllables significantly faster, than the McDaniel 
group. This difference in syllcible count also made for a 
faster total^count. The faster syllable count may have 
resulted from the more extensive ^practice which the Navy 
group had at Counting syllables prior to being tested on' 
these sample passages. 

Subsecjuent comparisons of speed using' the automated 
methods showed that the NAC counted words and syllables 
faster than the AFC, while the AFC was faster than the NAC 
in counting sentences. Inasmuch as the two groups were 
comparable in manually counting words and sentences, ^ is 
assumed that differences in automated word and sentence / 
counts are related to. the devices and not, to skill or 
practice differences between the -groups. A .definite con- 
cl'usion about differences in syllable count between the two 

16 
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devices cannot be made .from these data because of possible 
practice effects noted above for the manual ^lethod. , ^ 

It should also 'be mentioned that the NAC appeared to be 
more adaptable to the ^unique counting styles of the subject 
personnel. Sotne subjects pointed at each data point and 
Others tapped the stylus on the table. In addition, the 
subject personnel could easily stop counting midway through 
a passage, and resume counting later without losing the 
total count to that point 

Comparisons between the manual and automated methods 
showed that the AFC was significantly faster than the manual 
method at c6unting syllables. Thi? reduction in syllable 
count reduced the total time to analyze ti>e sample passages 
by 45%. The NAC was equally effective in reducing the speed 
of syllable counts, and- in addition the NAG was faster than 
the manual method ab caunting words. The NAC reduced total 
time ^46% over i:he njanuaX method, and also reduced total 
time by 20% over the AFC*, largely because of the syllable 
count differences whicli may have been related to a practice 
effect. ' • ' 

The above time savings can be readily converted into 

s • 

cost savings. The subject personnel were being paid ap- 
proximately $3.00 per hour to count sample passages from 
RTMs and NRCCs. If these personnel had counted the^^||^,000 
passages manually, it would have takren a total of 1,727 

12 



hours, and cost a total of $5,181. To do the same number of 

9 

passages using the NAC^. it would take 917" hours and cost 
$2,751. The savings resulting from use of the NAG would be 
$2,430. If the AFC had been used% it would have taken the 
subject personnef 1,082 hours, and cost $3,246. Total 
savings accrued from using the NAC compared to the AFC would 
be $4^5.00. As mentioned above, this savings may not be 
valid because of practice differences between the two 
groups. ' * 

Many measutes of reading grade levels depend on word 
sentence, and syllable counts. Of the approximately 40 
measures currently in use, at least half, including many of 
the most popular measures such as FOG, INDEX, Flesch Reading 
Ease Score, Forcast, Ride, and the Automated Reading Index, 
depend on these pr similar data points. These raeasiares ai^e 
often used in situations in which a large number of samples 
must be analyzed. As long. as these measures remain popular, 
automated methods (such as the NAC) which improve the speed - 
and reli2d:)ility of data point collection, will continue to* . 
be extremely us^^eful and cost effective. 

As a^^final note, although the NAC had been used primari! 
to count words, sentences, and syllables, it dbes have 
application in any situfltion in which simple "^counting is 
required. The device has been used to score responses on 
data sheets for research purposes, and is currently being 



\ 



used by the Naval -Education and j'riining Program Development 
Center to score s6me advancement e;?aminations , 
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standard E;rror>s •■dif the Means "arid- Ninety-Five 
Percent Conf idenc'e ifiterirals. ior» Gixiubs 1.-8 
d and 9-16; dirf'-syllabl^^T^ord/' and. sentfenpe 
^ count (Manual and. A*irto^'a.ted- iMethbdsJ.' ' 

MANUAL (PASgagE.. ir ' , 



Syllables 
Group 1-8' 

Group 9-16 

Wokdft 

Group ^1-8 

Group 9-16 

Sentennps 
* Group 1-8 

Group 9-- 16 



'Ninety*-f |.ve ^Percent 
Confidence Interval 



237.04 



2:?8.70 




NAC (PASSAGE II) 



Means 



Ninety^f ive ^Peroeni^ 
Conf-idence^ In€ervai 



^Syllables'^ " 
Group 1-8 ; 

Groilp 9-16 

words 

Group 1-8 

• Group 9-16 

Sentences 
' Group 1-8 

Group 9-16 



210.62 . 
210.62 * 

12i^.'50 
129l0Ot 

4.25 
*4.37 



210'. 0.1 
209.63 

129.04 
128.54^ 

3.82 
■ 3.95 



•211.23 
211.61 

li29.'96 . 
12 9. -4 6 

• 4.63 
' 4.79 



TABLE 3 



Comparison of Speed (in seconds) between the Manual and Automated 
Methods for the Navy and McDaniel Groups (Passages III~X)\ 



NAVY GROUP 



Manual 



NAC 



t-tests 



Syllables 
Mean 
S.D. 



272.42 
100.45 



113.38 
32.40 



11.96** 

df^l26 



Words 

Mean 
S.D. 



Sentences 
Mean 
S.D. 



120.30 




76.48 
17.59 



45.97 
18.1-5 



4.72** 

df=126 



ns 



Total Count 
Mean 
S.D. 



444.14 
168.51 



.235.92 
60.50 



9.23** 

df=126 



MCDANIEL GROUP- 



Manual 



AFC 



t-tests 



Syllables 
Mean 
S.D. 



357.34 
89.\34 



139.27 
31.39 



17.07** 
df=110 



Words 

Mean 
s'.D. 



IG'8.09. 
40.98 



104.30 
32.19 



ns 



Sentences 
Mean 

y s^,D. 



42.43 
23.25 



34.79 
18.22 



ns 



Total Count 
Mean 
S.D. 



506.'9€ 
127.65 



278. J5 
66.75 



11.77**- 
df=110 



** Indicates a significant difference between 'means at or belpw 
the iOl level. ' ■ 



TABLE 4 



Comparison between Navy and McDaniel Groups for Speed . 
(in seconds) of f^anual Method (Passages i, III-X) . 



Syllables 
Mean 
S.D. 



NAVY 
GROUP 



254.36 
100.83 



McDANIEL 
GROUP 



y\ 342.60 
^ 94.27 



t-tests 



5.39** 
df=141 



Words 

Mean 
S.D. , 

• Sentences 
Mean 
. : S.D. 

Total Count 
^ Mean 
^ S'.D. 



114.02 
67.26 



46^0 
26.22 



414.78 
.167.28 



105.17 
40.17 



40.86 
22.78 



467.84 
132.-34 



ns 



^ ns 



2,90** 
df=141 



TABLE 5 

« 

Comparison between Navy and McDaniel Groups for Speed 
{in seconds) of Autromated Methods (Passages II-X) . 



Syllables 
Mean 
S.D. 

Words 

Mean 
S.D. , 

Sentences 
Mean 
S.D. 

Total Count 
Mean 
S . d' . ^ 



NAVY 

GfiQUP ' (NAC) ' 

102.81 . 
36.22f 



69.46 
21.45 



41. as 

19.59 



213.32 
71.42 



McDANIEL 
GROUP (AFC) 

132.59 
35.22 



99.59 
33.51 



33.38 
18.09 



265.55- 
73.15 



t-tests 

4.95** 
df=141 



6.21** 
df=141 



2.43* 
df=141 



4.25** 
df=141 
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* Significant differences between means at or below the .05 level 
** Significant differences between means at or below/ the .01 lev^ 
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APPENDIX A 

Descri'ption of the 

1/ 

AU(tomated Flesch Count 
(AFC) , 
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2.5 




1 

AUTOMATED FLESCH COUNT 



The Automated Flesch Count apparatus consists of a 
Lafayette Instrument Coihpany Model No. 15019 Electronic, 
Metronome, ar Lafayette Ijistrument Company Model No. 58004 
Triple Date Recorder, and a master, control box. * - 

The metronome' is controlled by a combination rate . 
control and on-off switch. The metronome produces a ^constant 
rate ofv clicks that depends on 'the setting of the rate ' ^ 
control^ The rate of clicks may be varied from ^0 beats 'a 
minute to apprpximately 208 beat^^per minute. ^ 

The master control box selects the proper element to bt^ 
counted,; i.e., syllables, words, or sentences. A hand~hel<J 
switoh 
dount ar 



, — w — f — ^ f f 

s connected to the rtiaster control' box to begin the 
w.. d end the tSount. A tbggle switch in the lower right 
hand coi net' of the master i control box is used tq control the 
hand-help switch.' The hand-^eld switch is a push-on/push- - 
off type^^witch. Its operation^is similar to a retractable 
ball poinr pen. In the lower left hand corner, is a red 
pushbuttdn .switch labeled SENTENCE COUNT. Thi^ switch will 
perinili the ope'rator to , manually count elements. Each time 
the sw:i^tch is depres'sed /he selected display will advance by 
one. • . ' / 

The triple display counter has three 4 digit displays. 
Ttiey. are labeled as' SYLLABLES, WORDS, AND SENTENCES. The ^ 
selection of the display , being used, is determined by selec- 
tion on the master /control box.^ * To the right of each 
display counter ris a ,push button. Depression of this push 
button, resets thit' display to zero. Each display is in- v. 
dependently reset. An on-off switch located on the triple 
display counter provid,es pow^r to the unit. A red indicator, 
light glows » when the equipment is turned on. . 



This description provided by W. McDaniel in "Inter-AnalySt 
Reliability and Time Measures for Automated and Manual 
Flesch Counts'" {'Reference^ 4) 
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APPENDIX 
Specifications for the 
Navy Automated Counter 
(NAC) 
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